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ABSTRACT 

This study was designed to develop effective and time 
efficient assessment techniques for self-appraisal of science text 
and science reading and to assess middle grade students' 
metacognitive knowledge about science text and science reading. 
Standardized interviews, word associations, concept maps, and 
objective examinations were administered to grade 5 students. 
Preliminary results of the interviews reveal that grade 5 students 
have reasonable declarative, procedural, and conditional knowledge 
about science text and science reading but much growth is needed for 
them to become efficient, successful science readers. These results 
identify metacognitive dimensions regarding science text and science 
reading that would be strengthened by explicit instructions designed 
to improve students* knowledge of the use of science text. Reciprocal 
teaching was also identified as a promising instructional strategy 
that provided the support required to develop self-appraisal skills. 
A list of 64 references is included and structured interview 
protocols, structured tests for assessing self-appraisal of science 
reading and science text, a form for concept mapping, and a set of 
word associations are appended. (CW) 
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Introduction 

A historical review of science curricula reveals major philosophical changes 
reported in the literature: the 1940*s dominated by textbooks with a natural history 
and science applications focus, the 1950*s with traditional content fcoussed texts, the 
1960's with inquiry based science, the 1970's with reactionary reflex to a content 
discipline approach, and the 1980's with a sdence-technology-sodety (STS) focus. 
Throughout these fifty years little has changed in the majority of elementary 
classrooms regarding science instruction — science is infrequently taught, textbooks 
dominate the instructional decisions, and biological topics represent the major area 
of consideration. The 1990's present an excellent opportunity to drastically 
influence science instrucaon in the elementary and middle school classroom. 
Numerous factors-like the STS movement, AAAS 2061 proposal, research results 
regarding construction of knowledge, interactive-constructive model of reading, 
cognition and metacognition, and thirty years of successful inquiry-oriented 
instruction-provide a broad foundation on which to plan change. 

Background 

According to Holliday (1984), reading is one of the most firequently used 
methods for ^^cience instrucdon. The instructional sequence is dictated by the 
textbook. The typical sequence is assign, recite, test, discuss test (Yager, 1983). He 
also stated that most instrucdon in science centers upon the mastery cf unique 
words and terms with little attention to meaningful understanding. Harms (1981) 
and Stake and Easley (1978) pointed out that 95% of science teachers use textbooks 
90% of the time. Pratt (1981) agreed that elementary school science instruction is 
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driven by textbooks. With such extensive use of textbooks for science instruction, 
several questions beg to be considered (Yore, 1986a). What do science teachers know 
about the reading process generally, science reading specifically, and science text? Do 
teachers use this knowledge in designing instruction? Do science teachers know 
what their students know about sdence reading and science textual t'" trials? Do 
sdence teachers provide explicit instruction to increase sdence reading skills and 
knowledge about sdence reading and sdence text? 

Generally speaking, elementary sdence teachers' views of reading coindde 
with current approaches to reading (Shymansky, Yore & Good, in press). Reading is 
no longer viewed as a matter of extracting meaning from print. Comprehension is 
no longer regarded as a myriad of subskills. Factors influencing comprehension 
may be inside the head and outside the head (Samuels, 1983; Schell, 1988). Inside 
the head factors indude prior knowledge, interest, ability, strategy use, linguistic 
competence, and goals. Outside the head factors indude text topics, text structure, 
and situational context. If Cie factor is changed, so is comprehension. 

Reading is presently portrayed as interactive/ transactive (Guthrie, 1981; 
Langer & Smith-Burke, 1982; Spuo, Bruce & Brewer, 1983) and constructive (Langer, 
in press; Rumelhart, 1980; Spiro, 1980a). Readers interact/ transact with text and 
context to construct meaning. The context includes the situation, the linguistic 
context, interest, and attitude. Valenda and Pearson (1987) viewed readers as active 
learners who use clues from text in concert with their prior knowledge, 
environmental dues, and social context to construct meaning. 

The present view of reading focusses on how the reader builds meaning from 
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print Comprehension is considered to be an on-going process of understanding 
that grows and changes as the reader proceeds through the text (Collins, Brown & 
Larkin, 1980). According to Shymansky et al. (in press), the interact! ve-constiuctive 
model of reading "encapsulates Piaget's iniportance of experience, Vygotsky's 
importance of language, and Karplus* importance of exploration, invention and 
application" (p. 3). 

Although elementary science teachers generally agree with the current view 
of the reading process, Shymansky et al. (in press) indicated that litfle was known 
about teachers' attitudes toward and knowledge about science reading instruction. 
They suggested that teachers know little about the imique features of expository 
text. Yore and Denning (1989) found little evidence in secondary science classrooms 
of informed science reading instruction or sdence textbook utilization that reflected 
current models of reading. 
Sdence Fducation 

Most sdence educators would agree that elementary and middle school 
sdence learning involves an active exploration of conaete events by students who 
in turn construct understanding of these events utilizing a variety of intellectual 
processes. Sdence teaching should fadlitate this process by engaging 
misconceptions, providing meaningful experiences, guiding the constnjction 
process with a supportive scaffolding, connecting new constructions with recorded 
discoveries, and developing requisite skills, abilities, and processes. Generally, 
effective sdence teaching can be considered as repladng one misconception with 
another conception, where the second conception is more accurate than the first 
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misconception. 

Meta-analysis of the 1960 science curricula research results indicated that the 
learning cycle approach was effective in improving students' achievement, 
thinking, and process skills (Shymansky, Kyle & Alport, 1983). Yore (1984, 1986b, 
1986c) found that structured inductive and semi-deductive inquiry could be 
successfully used with specific groups of elem.entaiy school learners. Lawson, 
Abraham, and Renner (1989) reported that the structure and sequence (explore, 
invent, apply) of tihe le^niing cycle were critical. Shymansky and Yore (1980) 
reported fliat science instruction should provide a problem focus, a concrete 
experience, a scaffolding for construction of imderstanding, and opportunities to 
enrich, expand, and reinforce learning. 
Science Text 

Reading a text book is an integral part of most science classes. Unfortunately, 
factors related to both reader and text make the imderstanding of science textual 
materials a difficult imdertaking for many students. Because of repeated exposure to 
narrative text (fiction) and because narrative text has a consistent text structure 
(setting, beginning, development, ending), students develop a schemata or 
cognitive fi-amework that helps them to construct meaning from narrative text 
(Stein & Glenn, 1979; Stem & Trabasso, 1982). Students have not been exposed to 
the same quantity of expository (subject matter) text. In addition, expository text 
does not have a consistent text shructure (Mulcahy & Samuels, 1987). The structure 
of science text can be description, collection, compare-confa-ast, causation, or problem 
solution (Meyer & Rice, 1984; McGee & Ridigels, 1985). Besides possessing an 
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tinfamiliar and inconsistent text structure, science text exhibits other unusual 
features, such as unique lexicon, semantic features and syntax, whidh lead to 
students* difficulties in understanding science text (Eisenberg, 1977; Flood, 1986). 
Expert learners ai e aware of the purpose of expository text, \ls grammatic features, 
structure and adjunct devices, and use them appropriately in the reading process 
(Brown, 1982; Samuels, 1983; Steward & Tei, 1983). However, many students do not 
have or use this knowledge (Hood, 1986). 
Science Reading 

Little research has been done on science reading, especially in elementary 
school (Weidler, 1984; White & Tisher, 1986). Although the research has been 
sparse, several conceptions of reading in science have been proposed. Wittrock 
(1981) applied a generative model of reading in which readers generate meaning by 
constructing relationships between their prior knowledge and the text to achieve 
sdence learning. Holliday (1988) modified Bransford's (1979) conception of reading 
to describe science reading. 

[Insert Figure 1 About Here] 
Holliday suggested that learners construct meaning through an interaction of critical 
tasks (the learner's goals), characteristics of the learner, and extracting ixiformation 
from the situation (text, charts, pictures, other people) while monitoring, 
strategically planning, and evaluating the process. The monitoring, strategic 
planning, and evaluation of the process are referred to as metacognition. 
Shymansky et al. (in press) proposed that "science reading involves the ' 
orchestration of concrete experience, prior knowledge, cognitive abilities, 
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metacognitive operations and language to create understanding" (p. 4). 
Unified Concep tion 

Little evidence has been foimd that indicates elementary school teachers and 
secondary science teachers actually apply what is known aboi * reading in their 
science instruction which utilizes textual material (Shymansky et al., in press; Yore, 
in press). Teachers generally value content reading instruction and content reading 
skills while not fully understanding the interactive-cons' -uctive reading model and 
metacognition. Yore (1987) explored the potential of utilizmg science text embedded 
in science inquiry and the mediating effects of general reading ability. He found that 
middle school students were afforded greater achievement equality when reading 
assignments followed experience and group discussion to invent a concept. 
Realizing the full potential of text involved skill development, supportive 
scaffolding, knowing when, why and how to use text material, accessing prior 
knowledge or providing prior experience. 
Prior Knowledge 

Prior knowledge is a critical factor in constructing laeaning. Prior knowledge 
is knowledge and vmderstanding constructed from previous experience. It is usually 
described as organized ^ id stored in structures called schemata (Rumelhart, 1975). 
Prior knowledge u iudes conceptual knowledge (knowledge of concepts) and 
metacognitive knowledge (knowledge and control of understanding and the reading 
process). 

Conceptua l Knowledge . Readers use their prior knowledge to interpret, 
understand, and recall text (Bransford & Johnson, 1973; Mulcahy & Samuels, 1987; 
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Spiro, 1980b). Pearson and Johnson (1978) characterized the construction of 
meaning as building a bridge between new and prior knowledge. According to 
HolUday's model of science reading, the understanding students build is based on 
the students* prior conceptual knowledge in interaction with other factors. 

The amount of direct experience with the concept to be understood has a 
tremendous influence on what readers bring to and take away from the science text 
(Osborne & Wittrock, 1983). Learners do not come to new learning with no 
background knowledge or experience. Long before students are formally taught, 
they construct conceptual knowledge based on past experiences, current experience, 
and their interactions. This prior conceptual knowledge is not a collection of 
isolated ideas; it is a conceptual framework that forms the student's view of the 
world. Driver and Erickson (1983) referred to student prior knowledge as invented 
ideas and suggested these conceptual frameworks are strongly held and resistant to 
change. Unfortunately these conceptual frameworks may be different than the 
desired reality. When new knowledge is compatible with prior knowledge, 
understanding is much easier than when new knowledge and existing schemata are 
not compatible (Lipson, 1984; Maria & MacGinitie, 1987). The amount of prior 
knowledge (Recht & Leslie, 1988) and the appropriateness of prior knowledge 
(Lipson, 1984) influence both understanding and the amount of information 
recalled from text. Although there appears to be agreement that prior conceptual 
knowledge plays an important role in comprehension, Shymansky et al. (in press) 
pointed out that "the role of prior knowledge or naive theories in science Teaming 
was not well accepted by elementary science teachers" (p. 16). 
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Metacognitive Knowledge. Readers' metacognitive laiowledge about reading 
is an important dimension of prior knowledge (Jacobs & Paris, 1987). Cognition is 
an interactive-constructive process, and metacognition is a conscious reflection on 
this process. Recent reading research has focussed on the role of metacognition in 
the reading process. Metacognition involves the awareness and control of oneself 
and the reading process. Baker and Brown (1985) and Torres t-Pressley and Waller 
(1984) pointed out that beginning and poor readers lack metacognition associated 
with reading. Some diildren do not realize that they can us<i tiieir prior knowledge 
in order to understand what tiiey read. On the otiier hand, some readers tend to rely 
too heavily on prior knowledge when faying to understand iext (Maria & 
MacGinitie, 1987). 

Jacobs and Paris (1987) divided metacognition into two broad categories: self- 
appraisal and self-management of cognition. Self-appraiseJ of cognition includes 
knowledge about aspects of reading (declarative knowledge), knowledge about how 
to use specific proceuures during reading (procedural knowledge), and knowledge 
about when and why to use these procedures (conditional knowledge). Self- 
management of cognition includes planning, monitoring, and evaluating cognition. 
Studies have focussed on knowledge students have about reading of narrative text 
(Meyers & Paris, 1978; Paris & Jacobs, 1984). The literature contains no studies about 
students' knowledge about science text and science reading. 

Given the wide agreement among researchers that prior knowledge 
influences understanding of text, it is surprising how littie research has been 
directed toward determining what the learner knows before instruction /egins 
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(Erickson, 1977). Since students' conceptions and misconceptions influence the 
development of valid new concepts, teachers must know and consider the state of 
students' prior knowledge when designing instruction. Roe (1987) and Hood (1986) 
pointed out that one of teachers' major tasks in helping children comprehend text is 
to determine their background knowledge before asking them to read expository 
text. Engel Qough snd Driver (1986) stated: 

From an educational perspective it has been argued that it 
may be necessary to take accoimt of the ideas and beliefs 
that yoimg people bring to their formal study of sdence if 
these ideas are to be .successfully modified by instruction, 
(p. 473) 

In determining how to assess prior knowledge. Holmes and Roser (1987) 
suggested the aiteria of effectiveness and efficiency. Effectiveness refers to the 
amount of information obtained in relation to the time it takes to prepare the 
assessment , and efficiency refers to the relationship between the amount of 
information obtained in relation to the t?me it takes to give the assessment. 
Individual interviews aie frequentiy considered to be the most effective assessment 
technique; however, they tend to be very time consuming to administer. Several 
researchers have suggested other approaches that may be more effident and still 
have similar effectiveness, such as objective tests, concept maps, and word 
assodation ( Jonassen, 1987; Lim, 1989; Roe, 1987). 

This study attempted to compare the effectiveness of several assessment 
techniques and to consfanct a valid and reliable objective instrument. This 
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preliminary report will concentrate on the assessment of metacognitive knowledge 
about science text and science reading. The assessment of conceptual knowledge 
about plants and science processes and the metacognitive knowledge of the scientific 
enterprise will be reported later. 

Research Focu5? 

The following fod guided the design, development, and conduct of this 
investigation. 

1- Can the Jacobs-Paris approach be utilise to develop an 
instrument to assess self-appraisal of science text and science 
reading? 

2. Which techniques of assessing self-appraisal of science text and 
science reading (structured interviews, structured objective tests, 
concept maps or word associations) provide acceptable levels of 
effectiveness while being efficient in a classroom setting? 

Procedure 

The following study was conducted in order to determine both the 
effectiveness and efficiency of several assessments designed to determine students' 
prior knowledge of science concepts^ science reading, and science text. A 
combination of structured interviews (SI), structured tests (ST), concept maps (CM), 
and word associations (WA) was used to assess student?;* prior knowledge of science 
ideas, science reading, and science text and to judge the effectiveness and efficiency 
of alternative designs in relation to structured interviews. 
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Instruments 

Structured Interviews (SI) , The student and researcher sat together at a table 
in a quiet location free from distractions. Twelve questions related to the target 
topics (science ideas, science reading, science text) were asked in a nonjudgmental, 
supportive manner, Ea^Ji student was asked a selection of six metacogmtive 
questions (science reading and science text) and six consistent conceptual questions 
(plant kingdom and science processes). The metacognitive questions were 
geiiciated to reflect the self-appraisal of an efficient, successful science reader (Jacobs 
& Paris, 1987; Yore & Denning, 1989). Six different interview protocols were 
utilized. Interviews lasted from 9-13 minutes and were tape recorded. 

The questions were developed to parallel the metacognitive and conceptual 
dimensions reflected in the requisite knowledge about science text and science 
reading, and the concepts and processes involved with the study of plants. Some 
questions related to topics and had similar form to questions used in the structured 
test to allow direct comparison. Other questions considered topics or processes not 
included m the structured tests but were included because they utilized the full 
potential of the interview technique. 

The interviewer probed for rationales to responses and clarified vague or 
uncertain responses. The interviewer avoided prompting the students, but efforts 
were taken to insure that the students' metacognitive and conceptual knowledge on 
specific issues was fully assessed. The sbc stractiu-ed interview protocols are 
provided in Appendix A, 

Structured Tests (ST) . Students were given a test of 30 multiple ±oice and 
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shoi' /esponse items (14 conceptual, 16 metacognitiveX Two conceptual items 
consisted of two-parts. The tests did not have a time limit and took the students 
from 10-33 minutes to complete. There were two forms of the test with common 
conceptual items and imique metacognitive items. 

The conceptual test items were selected from two forms of a test on plants 
associated with a currant sdence iextbook series. The decision to utilize the 
prepared test items was based on the decision to use the specific sdence series textual 
materials in future researdi. 

The metacognitive test items, utilizing a revised multiple choice format with 
an open response option, were developed to reflect the what, how, why, and when 
dimensions of sdence reading and sdence text (Jacobs & Paris, 1987; Meyers & Paris, 
1978; Paris, Cross & lipson, 1984; Paris &: Jacobs, 1984). Items to assess dedarative, 
procedural, and conditional knowledge related to specific factors regarding sdence 
text and sdence reading were developed from the desired image of an effident, 
successful reader of sdence materials (Yore & Denning, 1989). The pool items were 
randomly assigned to two forms utilizing a stratified approach. Each form assessed 
some aspect of metacognitive knowledge of each dimension identified. Neither 
form assessed all factors or all three aspects of self-appraisal of a g./en factor in the 
desired image. Form A and Form B of the structured test are provided in Appendix 

Concepts Maps (CM). Students constructed two concept maps, one of a 
sdence topic and one of a sdence reading. The assessment technique included a 
brief review of concept mapping with students utilizing an example concept from 

o 14 
ERLC 



14 



language arts. Next, students were provided two sheets of paper with the target 
concept (science reading and plant kingdom) dearly indicated on the center of the 
page. The concept maps took from 8-20 minutes to complete. The response sheets 
for the concept maps are provided in Appendix C. 

Word Associations (WA). Students were given a list of repeated topics and 
asked to write words they believed were associated wifli the target topic on the line 
provided. Students were encouraged not to knowingly repeat themselves. The 
specific target topics were science reading, science text, and plan! kingdom. The 
word associations took between 10-12 minutes to complete. The response sheets for 
the word associations are provided in Appendix D. 
Sample 

The study was conducted in four elementary schools in a small dty in the 
interior of British Coltimbia, Canada. The schools represented a wide range of sodo- 
economic areas, rural/urban setting, and school and dass sizes. The sample for the 
study induded 83 grade five students (n = 24, 21, 23, 15). The gender distribution 
was 40 males and 43 females. The average age of the students was 11 years 
3 months, and the age range was 9 years 11 months to 12 years 5 months. All of the 
students spoke English as their first language. 

The school district was well known to the research team. It had a well- 
developed language arts program and instructional program to enhance thinking. 
The science program utilized a traditional textbook series enhanced with hands-on 
experiences. The selected schools represented the full range of educational 
programs and student population in the school district. 
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Data CcUecHon 

The data collector was a knowledgeable, experienced teacher and former 
consultant in the school district. The study was carried out on four consecutive 
school days in early June. The assessments were administered on one day in each 
school. The four assessment tasks were administered in the same sequence and 
general time frame at each school. The sequence for the tests was ST, SI, and WA 
or CM. There was a minimum of 60 minutes between each assessment task with a 
physical activity to reduce transfer, and no science or reading instruction was 
provided tmtil all tasks were completed. Students in each dass -v/ere randomly 
assigned to ST Form A or Form B. They were again randomly assigned to WA or 
CM. Six students were randomly selected to represent a stratified sample of above 
average, average or below average male and female readers from each dass. The 
students were interviewed in an order determined by random assigiunent. The 
specific form of Hi^. SI protocol was determined by random assignment. 
Scoring 

Scoring for the metacognitive items of the structured interview was based on 
a scale developed by Jacobs and Paris (1987). Responses were scored 0 if they 
indicated no knowledge or incon-ect knowledge about sdence text and science 
reading; responses were scored 1 if they contained partial, vague or surface 
knowledge; responses were scored 2 if they contained predse, comprehensive or 
in-depth knowledge. The open response option in the structured test allowed 
students to write in their own response if none of the responses provided seemed 
appropriate. These responses were scored using the same criteria as those used for 
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flie predetermined responses. Form A or Form B composite scores were 
determined by the sum of the 16 individual item scores. The maximum composite 
score on Forms A and B was 32. 

The structured interviews' responses were scored utilizing the same criteria 
and 0, 1, 2 scale for individual items as used on the ST responses. The total 
composite score for the SI was the sum of the scores on individual items contained 
in the specific protocol used. Composite scores are only meaningful for comparison 
amongst tlie four students exposed to the same protocol, otherwise th^y simply 
indicate a global metacognitive assessment on a 0-12 scale. 

Word Assodatioris were scored utilizing the same criteria and procedure for 
the open responses in the ST and SI. Each association was judged regarding its 
validity and the degree of imderstanding it conveyed. If several words were 
examples of one proposition or concept, ihey were grouped together and assigned a 
score based on the criteria for open responses in ST and SI. The scoring procedure 
for the WA is illustrated by the following science text example: 

Science Text: Part of a book 

Science Text: Table of Contents 

Science Text: Chapters 

Science Text: Glossary 

Science Text: Index 

Parts of a book (scored as 1) followed by a list of table of contents, chapters, glossary, 
and index (clustered and scored as 1). The composite score for the WA was 
determined by summing the points awarded each association provided. The 
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composite scx)re indicated a global metacx)gnitive assessr.ient of the target idea. 

Concept Maps were scored according to established examples, propositions, 
hierarchy, and cross links criteria (Novak & Go win, 1984, p. 35). The specific scoring 
criteria were: 

1- Propositions, The valid meaning relationship between two concepts 
indicated by the connecting line and linking word(s) (score 1 point each). 

2. Hierarchy, Subordinate valid concepts are more specific and less general 
than the concept drawn above it in the context of the material being 
mapped (score 5 points for each level of the hierarchy). 

3. Cross Links. Meaningful, significant and valid connections between one 
segment of the concept hierarchy and another segment (score 10 points 
for each cross line that is both valid and significant or 2 points for each 
cross link that is valid but does not illustrate a s)mthesis between sets of 
related concepts or propositions). 

4. Examples. Specific events or objects that are valid instances of those 
designated by the concept label (** core 1 point each). 

The composite score of the CM was determine J by the total points awarded for 
propositions, hierarchy, cross links, and examples. The composite score indicated a 
global metacognitive assessment of the target concept mapped. 

Analysis of Data 

The analysis of data involved the validation of the ms(ruments, establishing 
measures of internal consistency, and the generation of descriptive statistics for the 
sample assessed. Validity was explored in three ways: face validity, construct 
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validity, and factor analysis. A fourth method of structural equation modeling is 
not complete at this time. Reliability was explored utilizing inter-item correlations, 
item-total score correlations, and a measure of internal consistency. The results of 
student performance are reported in percent frequency distribution, means, and 
standard deviations for individual items and total scores. 
Wiidity 

Face validity was engineered into the development by utilizing accepted 
constructs as the foundation for the instruments. The desired image of an efficient, 
successful reader of science materials was well grounded in the reading and science 
education research literature (Yore & Denning, 1989). The 20 factors of the desired 
image dearly describe many of the critical attributes of the ideal science reader as 
envisioned at this time. The blueprint of the ideal science reader was further 
clarified by expanding each of the 20 factors utilizing the self-appraisal dimension of 
metacognition to specify declarative, procedural and conditional knowledge 0acobs 
& Paris, 1987). The blueprint became a 20 x 3 matrix that served as the contingency 
table for designing the assessment instruments. 

The assessment strategy of the ideal science reader was based on established 
data collection techniques. The structured interview protocols sampled critical cells 
in the 20 x 3 matrix. It was believed that the rich nature of the information 
produced by the interview format (six strategic questions for each protocol and the 
six alternative protocols) would allow assessment of target factors and, considered in 
combination, would provide a reasonable global assessment of metacognitrve 
knowledge about science reading and science text. A similar rationale was used to 
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generate a pool of 31 objective items. Two stratified random samples of 16 items 
(one critical item was common to both forms) allowed data to be collected on many 
factors of the ideal science reader by each form of the structured test. Not all factors 
by categories of self-appraisal were assessed. The concept map and the word 
association instnmients were designed to provide global assessments of 
metacognitive knowledge about science reading and science text 

Construct validity was partially explored by correlating the data collected by 
each method. The structured interview data were identified as the reference 
construct The quantitative data on individual items were correlated vdth data 
from similar items on the structured test The total scores on the interview were 
correlated with the equivalent combined scores of items from the structured test. 
The total metacognitive scores from the structured interview, the total score from 
the structured test, the global scores from the concept map, and the global scores 
from the word' association were correlated to provide global indications that the 
instruments were assessing similar constructs. 

[Insert Table 1 About Here] 
The small but positive correlations between the interview data and the 
structured test item data and the structured test sub-scores were promising. 
Although only a very small amount of common variance was contributed by the 
shared assessment, improved item selection, item revision and combining the two 
forms into a single structured test may improve the construct validity of the 
structured test. The negative correlation between the £1 data and Form B is 
problematic. The concept ma;> and the word assodation require further 
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consideration to explain the surprising correlation values discovered. 

Principle components from rotated factor analyses were used to explore 
whether the structured test items clustered about dimensions of the ideal science 
reader. Table 2 indicates the principle components of Form A and Form B. The 
principle components revealed did not duster exclusively within the categories of 
self-appraisal or along the factors of the ideal science reader. Several of the principle 
components could be labeled with a meaningful unifying concept. These results 
may be due to the small number of items and the small homogenous samples used. 

[Insert Table 2 About Here] 

Structural equation modeling is planned to explore the construct valid 
further. USREL 6 will be used to determine if the hypothesized constructs were 
assessed by the items in the structured tests and to determine if items designed to 
assess a specific construct tx>nverge on that construct. 
Reliability 

Form A and Foxm )5 were the only data investigated for reliability. Inter-item 
correlations of items measuring the same factor of the ideal science reader or the 
same category of self-appraisal were considered. The majority of the item dusters 
reveal positive but small correlations indicating that these target items were 
assodated as intended. 

The item-total score correlations were used to assess the contributions made 
by each item to the total assessment. The correlations reveal that all items were 
positively assodated with the total score of the structured test. Ten of the 16 items 
of Form A and 8 of the 16 items of Form B were significantly correlated with the 
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spedfic total score. 

The Alpha measures of internal consistency were low (Form a = 0-44 and 
Fonn B = 0.26). These low measures indicate thai some individual items need 
revision and the number of items in the test should be increased. Preliminary 
exploration of excluding 3 items iTom Form A and 6 items from Form B raised the 
internal consistencies of the remaining test items to 0.50 and 0.40. 

In summary, the structured tests appear to be an assessment technique that is 
efficient and has reasonable effectiveness. The concept map technique is pronusing, 
but familiarity with the process must be assured before using it ao an assessment 
technique. The word association technique used needs further refinements. The 
structured interview and performance-based assessment still appear to be the most 
effective assessment techniques, but their efficiencies are low. 
Results 

The results of this study must be viewed in the light of the validity and 
reliability reported earlier. Furthermore, the sample represented a school district 
that has spent considerable effort on erthandng students* thinking and reading 
albeit narrative text was emphasized. Nevertheless the results reported may 
provide insights for further research or sharpened focus of future enquiries. 

Interview Results. The 24 interviews indicated that the Grade 5 students: 

1. knew that pictures, diagrams charts and definitions 
help them to understand what they are reading, 

2. were aware of different aspects of text structure, 

3. knew that visual imagery and reflection were used in 
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order to understand what was read, and 
4. knew that understanding is the main goal of reading. 
Generally the students knew that they interact with text and other information 
sources in order to construct meaning. In addition many of them knew how, when 
and why to interact with text and other sources in constructing meaning from text. 
Eighty-five percent of the 140 valid responses indicated that the students had 
appropriate declarative, procedural and conditional knowledge about science text 
and science reading. Fifteen percent of the responses indicated siudents had 
inappiopriate, incorrect or no knowledge about science text or science reading. 

The number of responses iridicating knowledge about science text and 
knowledge about science reading were similar, 84% and 86%, respectively. An 
examination of ef^ch category of self-appraisal indicated that 89% of the responses 
designed to elicit conditional knowledge, 84% for questions of procedural 
knowledge, and 83% for questions of declarative knowledge were scored 1 or 2. 

Although there were only foui- responses for each question, the responses for 
individual questions or groups of questions merit consideration. Students who 
responded to questions about science text, what it represents and its purpose and 
value, indicated that text represents ideas and the words help the readers to 
understand. Some students realized that science text was not infallible and 
difficulties experienced with text could be due to text and not just the reader. Most 
students realized that some parts of a paragraph were more important than other 
parts and why, and that sentences were arranged in order depending on whether the 
paragraph outlined directions or had a main idea and supporting details. 
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Unfortunately students did not realize that since text has different structures it 
should be read differently. Students knew that adjuncts such as pictures, graphs and 
diagrams were useful as an aid to understanding and recall but were not sure when 
to refer to these aids. 

Students who responded to questions related to the goal of reading knew it 
was important to understand, knew how they knew when they did not imderstand 
and knew they should think about what they were reading. They also knew they 
should check to see if what they were reading made sense but did not know when to 
do so. Students knew that the purpose of self-questioning, re-reading, looking 
ahead and visualization was to facilitate vmderstanding of text. Th2y also knew that 
teachers, other texts, and other students could help them to imderstand. Although 
they knew they could imderstand text better with prior knowledge of the topic, not 
all students referred to prior knowledge as a source to aid in understanding. 
Students saw ability to understand as a characteristic of a good reader but only one 
student out of four equated application of strategies with being a good reader. 
Students suggested a variety of ways for decoding a word or understanding a 
sentence. These included use of context, re-reading, reading ahead, and consulting 
with other students and the teacher. Only one student mentioned "sound it out". 
Although students perceived a difference between simply reading and reading to 
remember, not all of them suggested writing information down as a way to 
remember. One student suggested making a tape and playing it at night. Although 
students knew why and when to skim text, and had a fairly good notion ofliow to 
skim, they had very little knowledge of how to summarize or find the main idea. 
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Students appeared to have no knowledge of first determining a purpose for reading 
and then proceeding bab. d on the demands of the task. But, they seemed to know 
what to do and hov/ and when and why to do it after they started reading. 

Structured Tests . The descriptive results for all assessment techniques and 
individual items are reported in Table 3. An inspection of the combined results for 
Form A and Form B indicates that the forms were not equally difficult. 
Performance on individual items revealed three general categories of results: high 
success (> 80% scoring 2, <10% scoring 0), average success (>80% scoring 2 or 1, <20% 
scoring 0) and low success (<50% scoring 2, >30% scoring 0). 

[Insert Table 3 About Here] 
Results from Form A indicated that high success was noted on items 15, 16, 
19, 21, 22 and 28. Low success was noted on items 29 and 30. Results from Form. B 
indicated that high success was achieved on item 23 and that low succes was noted 
on items 21, 23, 24 and 30. 

The central themes amongst the high success items consider the purpose of 
reading (A15), text features (A16, A22 and B23), prior knowledge (A19), and 
strategies (A21 and A28). The central themes amongst the low success items were 
text features (B21 and B24), reading to remember (A30 and B30), and strategies (A29 
and B22). Many similarities to the interview results were found regarding strategic 
planning, sununarizing and text struchire. 

Word Ass ociation and Concept Maps . The word association and concept map 
approaches provided some interesting global data, but to achieve consistency of 
scoring took a great deal of time. The final score produced was a global assessment 
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and did nof provide much more information than an intuitive scormg procedure. 
In both procedures the students appeared to simply brainstorm their responses 
vwthout considering relationships between the responses. It is difficult to determine 
if the results reflect: students' lack of organization of their knowledge of science text 
and science reading; their perception that science text and science reading are the 
same concept; lack of familiarity with the CM and WA procedure; or lack of 
familiarity with these particular formats. Further consideration is planned, but the 
two assessments produced similar responses. 

Im plications 

This preliminary study revealed that assessing readers' knowledge about 
science reading and science text will be a difficult process to bring the asse..cment 
techniques to psychometric standards of validity and reliabilily. The added issues of 
effecaveness and efficiency compoimd the problem. The solution will need to 
consider two levels of acceptance regarding classroom application and research 
applications. Each of the techniques described in this study has reasonable validity 
to identify readers' strengths and weaknesses. Each technique provided 
information to help guide general instruction, but none p. yvided the precision 
necessary to make critical instrMctional decisions or to be used in precise research 
studies. 

Future research and validation studies will need to revisii^ the desired image 
of a science reader, the conception of science toxt, and the conception of 
meracognition. The desired image of a science reader relied heavily on the general 
readi.,g research and recent research results v/ill help clarify the image. The 
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conception of science text was based on studies from the 1970s; these studies and 
recent reports do not fully agree on the nature of sdence text. Therefore the 
new results must be reflected in the assessment blueprint. The original conception 
of declarative, procedural and conditional knowledge about science reading and 
sdence text contained an assumption of hierarchy that was not confirmed by the 
grade 5 interviews. The ftiZ2y aspects of metacognition require darification 
regarding automatidty, application range, affective dimensions, and other issues. 

The strurtural equation modeling (USREL 6) may provide additional insights 
into the item-construct alignment. The fitting of various models and data may help 
darify any causal relationships. The stincturai maps will hdp sharpen the design of 
items and the sdection of the items. The combination of traditional test design 
procedures, structural equation modeling and construct darification will likdy lead 
to more effective and effident assessment techniques. 
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Figtire 1: Tetrahedral Interactive Model (Holliday, 1988) 
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Table 1: Correlations Between Assessment Techniques and Specific Components 



Instrument ST Items ST sub-score ST A ST B ST A/B CM WA total 
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0.16 


0.21 


0.07 


-0.21 


0.12 


0.07 


-0.30 


SI 
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0.04 


0.17 


0.41 
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0.30 


0.42 


0.16 
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Table 2: Principle Components Resulting From the Rotated Factor Analyses 
of the Structured Test Items (Loading Value) 



Factor 1 Factor 2 Factor 3 Factor 4 Factor 5 Factor 6 FactoFT 

Form A 27 (.718) 19 (.827) 23 (.793) 30 (.840) 28 (.776) 26 (.803) 15 (.904) 

17 (.690) 21 (.752) 29 (.470) 25 (.525) 20 (.659) 

18 (.679) 16 (.705) 
24 (.639) 

22 (.352) 



Form B 26 (.817) 23 (.813) 16 (.921) 
30 (.686) 25 (.746) 17 (.551) 
29 (.580) 18 (.421) 



21 (.828) 28 (.866) 15 (.929) 22 (.927) 
19 (.555) 20 (.375) 

24 (.305) 
27 (.296) 
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Table 3: Descriptive Statistics for Specific Assessment Techniques and Items 





Full Value (%) 


Partial Value ( 


:%) No Value (%) 
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SD 


SI (n=24) 








8.67 


1.93 


ST 












Form A (n=42) 








25.41 


3.28 


Item 15 (n=42) 


40.5 


9.5 
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1.91 


.30 


16 (n=42) 


92.9 


7.1 
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1.93 


.26 


17 (n=42) 


73.8 


19.0 


7.1 


1.67 


.61 


18 (n=42) 


83.3 
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11.9 


1.71 


.67 


19 (n=42) 


90.5 


7.1 


2.4 


1.88 


.40 


20 (n=42) 


71.4 


26.2 
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1.69 


52 


21 (n=42) 


85.7 


9.5 


4.8 


1.81 


.51 


22 (n=42) 


85.7 
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14.3 


1.71 


.71 


23 (n=42) 


35.7 
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14.3 
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.68 


24 (n=42) 
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11.9 


21.4 


1.45 


.83 
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57.1 
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1.29 


.60 
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11.9 
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.68 
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11.9 


1.71 


.67 
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33.3 
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26.2 


33.3 
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Form B (n=40) 








20.56 


3.49 
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Structured Interviews 
Protocol 1 

1. What is the main goal of reading? 

2. Why do you use the pictures, diagrams and definitions on a page in 
your science text? 

3. Where do you get the information to help you make sense of your 
science text? 

4. How do you skim a paragraph? 

5. What topic in science would you find easiest to understand? Why? 

6. Is it important to see if what you're reading makes sense? When 
should you do this? 

Protocol 2 



1. Are there things on the page of your science text that help you to 
understand what you are reading? What are they? 

2. Should you ask yourself questions while you are reading your science 
text? Why? 

3. Is there some part of a paragraph that is more important than the 
rest of the paragraph? Whaf is it? Why is it important? 

^. Should you read all paragraphs the same way? Why? 

5. Should you go back and read something again? When? 

6. How do you know when you don't understand what you have read? 

Protocol 3 

1. What could you do if you didn't understand what a sentence was about? 

2. Do you ever make pictures in your head of what you are reading? Why? 

3. What makes someone a good reader in science? 

4. Do you ever look ahead in your science text? Why? 

5. Is there something you should do before you start reading in your 
science text? What is it? 

G- If you are reading a science text to study for a test what could you 
do to remember the information? 
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Protocol 4 

1. Should you believe everything you read in your science text? Why? 

Z. How could you figure out a word in your science text that you 
didn't know? 

3. Are words in a science text just words? What is important about 
them? 

4. When would you skim a paragraph? 

5. Are there times when you don't understand what you read in your 
science text? What would you say is the reason you don't understand? 

6. When/How do you use the pictures, diagrams and definitions on a page 
in your science text? 

Protocol 5 

1- Is there some part of a paragraph that is more important than the 
rest of the paragraph? What is it? Why? 

2. How do you summarize what you have read? 

3. Are the sentences in a paragraph arranged in any order? How? 

4. Some words in science texts seem to go together. If you saw the word 
'if what word would you expect to see later? 

5. If you are reading a science text what could you do to remember the 
information? 

6. Should you think about what you're reading? Why? When? 
Protocol 6 

1- Do you do anything before you start reading your science text? What? 

2. What makes someone a good reader in science? 

3. What could you do to help you understand what you are reading? 

4. If a section in your textbook describes a problem what do you expect 
by the end of the section? 

5. Is there a difference between reading and reading to remember? What 
is the difference? 

6. How do you find the important (main) idea in a paragraph in your 
science text? 
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Appendix B 



Form A and Form B 
of Structured Tests 
of Self-Appraisal 
of Sdence Reading and Sdence Text 
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Form A 



Name: 



INTRODUCTION 

This test is to help me to understand what you know about the plant 
kingdom, science text-books and science reading. You will not be graded 
on this test, but you are taking part in a scientific study so do your 
besti Read the directions for each part before you sta.t. the questions. 

^ Science Text. books and Science Reading 
Read each question carefully and select ohe answer (circle letter) that 
best answers the question or write your own answer to the question in 
the space provided. 

15. The main goal of reading science text, is 

a. to say all the words 

b. to understand what it's about 

c. to read quickly without making mistakes 



d. 



16. You refer to the pictures, diagrams and definitions on a page 
in your science text because 

a. they help you to remember and understand 

b. you won't need to read the words 

c. they are interesting 

d. 

17. The infonnation that you need to make sense of science text is 

a. only in the text 

b. in the text and in your head 

c. in the teacher's head 
d. 
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18. Skimming is 

a. reading all t:he short words and not the long ones 
something poor reader's do 

c. a quick way to find out what the text is about 

d. 

19. Topics in scienc" are easier to understand 

a. when you know scmiething about the topic 

b. when the prijiting on the page is large 

c. vrfien you have a comfortable place to sit 
d. 

20. Things on a page of your science text, that are there to help 
you understand the words are 

a. page numbers 

b. pictures and diagrams 

c. sentences and paragraphs 
d. 

21. If you were reading science text you would ask yourself 
questions and think about what you were reading 

a. because it is a good thing to do 

b. to make sure that you knew all the words 

c. to see if you understood what you were reading 
d. 
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22. The first sentence in a pai'agraph 

a. is no more important than any other sentence 

b. starts the paragraph 

c. fi*equently tells you what the paragraph is about 

d. 

23 • Sentences in a paragraph are put together in different ways. 
Some describe, some are in order of happening and some compare 
and contrast ideas. Good readers 

a. read all paragraphs the same way 

b. read paragraphs differently 

c. read them all quickly 

d. 

24. Good readers go back and read things over again because 

a. it's a good idea 

b. they forgr^t some words 

c. they didn't understand what they had read 

d. 

25. If you don't know what a vrfiole sentence means you should 

a. read it again 

b. sound out all the words 

c. think about the other sentences in the paragraph 
d. 
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26 • Making organized mejital pictures of what j'ou read is useful 
because 

a. you won't need to write it down 

b. it helps you to remen±>er it 

c. it makes the t^xt mor^e interesting 

d. 

27 • Someone is a good reader in science if they 

a. read quickly 

b. like reading 

c. do things to help th^anselves understand 

d. 

28# looking ahead in your scieiice text is a good idea for 

a. getting finished your reading faster 

b. skipping over hard parts 

c. checking out your guesses 
d. 

29» Before I start to read 

a« I choose a comfortable place to read 

b. I stop and think about vA\y 1 am leading 

c. I don't make plans, I just start reading 


30* When I am reading ny science text to study for a test the most 
helpful thing for me to do is 
B. Outline what I have read 

b. write down everything I have read 

c. say all the words over and over 
d. 
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Fonn B Name: 



INTRDDUCTION 

Tliis test is to help me to understand wliat you know about the plant 

kingdom, science textbooks and science reading. You will not be graded 

on this test, but you are taking part in a scientific study so do your 

best! Read the directions for each part before you start the questions. 
PART U Science Textbooks and Science Reading 

Read each question carefully and select the answer (circle letter) that 
best answers the question or write your own answer to the question in 
the space provided. 

15- Text in science books is 

a. never true 

b. someone's view of what is true 

c. always true 

d. 

16. When you come to a word and you don't know what it means you 
should 

a. ask someone else 

b. use the words around it to figure it out 

c. go on to the next word 

d- 

17. The words in your science text are 

a. for you to pay attention to 

b. labels for ideas and experiences 

c. what the author guesses to be true 
d- 
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l8. You use skinning, key words, titles and headings 

a. when the text is difficult to read 

b. when you want to read quickly 

c. when you want an overview of the text 

d- 

19* If you don't understand what you have read it is 

a. because science texts are sometimes not well written 

b. always your fault 

c. always because you didn't use the right reading skills 

d. 

20 • You use the pictures, graphs and diagrams on a page by 

a. looking at them when your teacher tells you to 

b. looking at them \Aien you have finished the chapter 
c* looking at them vAien you need to 

d. 

21* The parts of a paragraph that are most important are 

a. the first and last sentences 

b. the middle sentences 

c. all the sentences 

d. 

22» To sunroarize information in science text you 

a. tell the in^xDrtant ideas 

b. tell all the facts 

c. tell the last sentence in the section 
d. 
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23 • The sentences in a paragraph are 

a* not arranged in any special order 
b. all arranged in the ^same way 
c* arranged in certain orders 

d, 

24* When you are reading science and you see the word "if", 
what do you expect later? 

a. a "then" 

b. a "maybe" 

c. nothing 

d. 

25* If you are reading your science text and you want to remember 
the information you should 

a. write it down in your own words 

b. say every word over and over 

c. think about remembering it 

d. 

26. Before you read your science text it is irrportant to 
a. think about what you already know about the topic 
h* find a ccMnfortable place to read 
c. make sure you can read all the words 
d. 
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27* Someone is a good reader in science if they 
a. read quickly 
like reading 

c. do things to help themselves understand 

d, 

28. To understand what you are reading you should 

a. ask yourself questions about the jjnportant ideas 

b. see if you know all the words 

c. keep reading 
d, 

29. When a section in your science textbook describes a problem, 
what do you expect later? 

a. another problem 

b. the cause of the problem 

c. a solution to the problem 
d. 

30. Is there a difference between reading and reading to 
remember? 

a. just reading is harder because you have to sound out all 
the words 

b. just reading is the same as reading to remember 

c. reading to remember is harder because you have to make 
notes as you read 

d. 
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Appendix C 

Concept Map 
for 

Science Reading 
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Please di-aw a mind map of "Science Reading" 



SCIENCE READING 
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Appendix D 



Word Associations 
for Science Textbook and Science Reading 



ERIC 



51 



51 

Words and Ideac That Go Together 

Write down as many words or ideas as you can think of to go with 
"Science Textbooks". 

Science Textbooks 

Science Textbooks 

Science Textbooks 

Science Textbooks 

Science Textbooks 

Science Textbooks 

Science Text,b<x>ks 

Science Text,books 

Science Textbooks 

Science Textbooks 

Science Text.books 

Science Textbooks 

Science Textbooks 

Science Textbooks 

Science Textbooks 



(Use the back of this sheet for other words or ideas) 
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Words and Ideas That Go Together 

Write down as many words or ideas as you can think of to go with 
"Science Reading" • 

Science Reading 

Science Reading 

Science Reading 

Science Reading 

Science Reading 
Science Reading 

Science Reading 

Science Readir;^ 

Science Reading 

Science Reading 

Science Reading 

Science Reading 

Science Reading 

Science Reading 

Science Reading 

(Use the back of this sheet for other words or ideas) 
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